[None][perf] optimize encoder-decoder PyTorch performance - #16706

Summary

• accumulate replacement encoder work while decode continues, then release encoder microbatches at the configured iteration or token threshold
• capture and replay encoder-forward CUDA graphs for encoder-decoder models using user-configured batch-size, packed-token, and sequence-length buckets
• capture mixed decoder CUDA graphs for iterations containing newly admitted context requests and ongoing generation requests; enable this optimization by default when the encoder and decoder graph configurations produce usable shapes
• prepare qualified mixed encoder/decoder batches in one native call using persistent host buffers
• reuse request IDs and sequence slots for stable single-beam greedy decode batches, copy only compact argmax results to the host, and detect EOS/length completion without the general finish-reason tensor
• preserve the general sampler path for draft decoding, beam search, log probabilities, sampling, biases, minimum length, stop words, bad words, and other excluded features

Why

BART continuous-admission serving repeatedly mixes replacement encoder requests with active decoder requests. The existing path launched small encoder batches, rebuilt mixed decoder metadata in Python, and reconstructed sampling/finish metadata on every greedy decode step. It also ran eligible encoder and mixed decoder work eagerly instead of replaying fixed-shape CUDA graphs.

These changes reduce per-iteration CPU launch and device-to-host overhead while keeping decoder generation active during encoder admission. The CUDA-graph capture layout is derived from the encoder runner's effective capture keys so padding and runtime replay use the same buckets.

User interface

For encoder-decoder models, encoder_cuda_graph_config=EncodeCudaGraphConfig(...) enables encoder-forward CUDA graphs and defines batch-size, total packed-token, and maximum sequence-length buckets. encoder_max_batch_size remains the hard encoder capacity and admission limit.

enable_encoder_decoder_mixed_cuda_graph controls the mixed-batch decoder optimization. It defaults to True but becomes effective only when both the encoder and decoder graph configurations produce usable capture shapes. Set it to False to retain separate encoder and decoder CUDA graphs while disabling mixed-batch graphs.

Performance

BART-large-CNN

This comparison uses the same deterministic natural-length CNN/DailyMail workload on both backends: 1,024 unique validation articles sampled without replacement using seed 0, encoder lengths of 93–1,022 tokens, and no truncation.

PyTorch improves throughput by 12.7–20.9% and reduces mean latency by 11.0–17.2% compared with legacy TensorRT.

FLAN-T5 Large

This comparison uses google/flan-t5-large and a deterministic 1,024-request Super-NaturalInstructions workload derived from the official allenai/natural-instructions default/test split. Inputs longer than 512 tokens and reference outputs longer than 128 tokens are rejected rather than truncated.

With encoder and mixed encoder-decoder CUDA graphs enabled, PyTorch improves request throughput by 11.8–57.8% and reduces mean latency by 9.4–36.6% compared with legacy TensorRT.

Definition: <task definition>

Input: <instance input>

Output:

Validation

• focused executor, warmup, model-engine, sampler, and LLM-args unit tests cover graph admission and replay, stream and TP synchronization, fixed sequence-slot staging, greedy completion, no-repeat-ngram fallback, and configuration validation
• BART and T5 continuous-admission integration tests verify encoder graph replay and mixed decoder graph replay; both are assigned to H100 PyTorch post-merge L0
• targeted BART/T5 CUDA-graph integration run: 2 passed
• BART-large-CNN and FLAN-T5 Large PyTorch/legacy TensorRT benchmarks at concurrency 8, 32, and 64 on deterministic 1,024-request workloads
• pre-commit formatting, lint, safety, test-list, and type-analysis hooks
• DCO commit-message hooks

Dev Engineer Review

• Adds asynchronous encoder scheduling with microbatch thresholds, deadlines, CUDA events, and tensor-parallel synchronization.
• Adds encoder and mixed encoder-decoder CUDA graph capture and replay.
• Adds persistent host-buffer input preparation and native bindings for encoder-decoder batches.
• Adds fixed sequence-slot reuse and a fast single-step greedy sampler path.
• Preserves the general sampler path for unsupported decoding features.
• Adds and validates encoder_cuda_graph_config and enable_encoder_decoder_mixed_cuda_graph.
• Updates documentation, supported-model guidance, API manifests, and configuration tests.
• Main risks are asynchronous stream coordination, staging-buffer lifetime, mixed-graph key selection, encoder fallback behavior, and fast-path eligibility. Targeted tests cover these areas.

QA Engineer Review

• Adds executor tests for encoder warmup, admission boundaries, fallback scheduling, asynchronous completion, tensor readiness, and tensor-parallel publication.
• Adds model-engine tests for deferred two-pass warmup and fixed-sequence-slot staging and output restoration.
• Adds sampler tests for no-repeat-ngram fallback and greedy EOS/length completion handling.
• Adds TorchLlmArgs tests for encoder graph configuration validation and mixed-graph settings.
• Adds BART and T5 continuous-admission tests for encoder and mixed decoder graph replay.
• Updates CI test lists:
  • l0_h100.yml: adds BART and T5 continuous-admission tests.
  • l0_dgx_h100.yml: adds the two-GPU BART continuous-admission test.
  • l0_l40s.yml: removes the superseded T5 mixed-context test.
• The new integration tests are covered by H100 CI entries. Unit tests do not require test-list entries.
• Verdict: sufficient.